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Abstract 

The Science Notebook in a Universal Design for Learning Environment (SNUDLE) is a 
digital notebook that uses the Universal Design for Learning framework to support active science 
learning among elementary school students, particularly those who struggle with reading and 
writing or are unmotivated to learn science. Preliminary findings from the first of a two-year 
randomized control trial suggest no significant impact on motivation or academic achievement in 
science among the full sample of fourth graders receiving the SNUDLE intervention. Moderator 
analysis indicates significant positive interaction effects of the intervention on motivation in 


science and science content assessments among students with learning disabilities. 


Objective 

Achieving the vision of Next Generation Science Standards (NGSS) necessitates 
developing new approaches and tools to support students’ active science learning.!* This is 
especially true for students who have learning and other disabilities; who struggle with reading 
and writing; who have difficulty with memory, executive function, and learning strategies; or 
who have low motivation for science learning.*-’ The Science Notebook for a Universal Design 
for Learning Environment (SNUDLE) was created to help students, particularly struggling and 
unmotivated students, better realize the benefits of science notebooks. Universal Design for 
Learning (UDL) was chosen as the SNUDLE design framework to minimize construct-irrelevant 
barriers to learning and provide just-in-time supports for active science learning and effective 
science notebook use.*” 

This paper describes the preliminary findings from Year 1 of a 2-year efficacy study 
addressing the following research hypotheses: (1) compared with fourth-grade students using 
traditional paper-based science notebooks in business-as-usual classrooms, students in SNUDLE 
classrooms will significantly increase their science content knowledge and motivation in science, 
and (2) a subsample of students with disabilities will experience significant positive interaction 
effects of the SNUDLE intervention. Additionally, this study contributes to the body of UDL 


literature by rigorously examining applied UDL features and principles. 


Background 

There is clear evidence that students with disabilities can effectively participate in active 
science learning and that active science learning may promote better outcomes than textbook- 
based methods.'°:'!! Yet active science learning, which incorporates hands-on, direct involvement 


in science investigations and collaboration between students, requires students to simultaneously 


develop and use several complex skills (e.g., read, use math, build content knowledge, apply 
scientific reasoning), and this presents challenges for teachers supporting students with 
disabilities in their science classrooms.!*!> 

The research literature indicates that science notebooks can be used to support active 
science learning and the development of scientific literacy.!° '’ As a paper-and-pencil tool, 
however, the traditional science notebook presents a barrier to students who struggle in the 
learning process because students must be relatively proficient in reading and writing to use it. 
Further, teachers typically and use science notebooks primarily in a mechanical way—to record 
data, procedures, or definitions—and rarely to support the development of deep understanding 
through the active science learning process.'*!? Given these challenges, teachers need evidence- 


based tools and strategies to provide all students with access to the general curriculum through 


active science learning to meet the common set of high academic standards in NGSS.”° 


Meeting the Challenge of Active Science Learning 

Active science learning has the potential to improve elementary science learning 
outcomes for students with disabilities, with effective guidance.?!”? The suite of accessibility 
features to reduce construct irrelevant barriers, learner-specific flexibility, and concept 
scaffolding recommended by UDL principles can help provide this guidance and make active 
science learning more accessible to struggling students. In SNUDLE, the developers used UDL 
to take advantage of digital formats to design a new kind of science notebook. The basic premise 
of UDL is that barriers to learning occur in the interaction of students with curriculum, not solely 
in the capacities of the learner.*?> Therefore, universally designed tools offer a range of options 
for learners to access and engage with learning materials and are designed to accommodate the 


widest possible range of learner needs and preferences.°? 


Technological advances made the development of UDL approaches, texts, content 
curricula, strategy-based interventions, and assessment possible.?°?* UDL digital environments 
provide the infrastructure and flexibility necessary for the creation of accessible, highly effective 
settings where students are actively guided through the process of constructing meaning by the 
provision of just-in-time feedback and contextual supports that can be gradually withdrawn as 
students’ expertise increases.”?*? This design approach supports inclusive classroom teachers 
with flexible tools to create more effective and differentiated learning experiences for students.** 

SNUDLE Overview 

Like traditional science notebooks, SNUDLE provides students space to collect, 
organize, and display observations and data; space to reflect and make sense of inquiry 
experiences; and multiple opportunities to demonstrate understanding at every stage of the 
investigation through text answers, data tables, photo uploads, and drawing tools, and receive 
formative, interactive teacher feedback on their work. However, with UDL as the design 
framework*> and digital technology as the platform, SNUDLE differs from traditional science 
notebooks in several key ways (Figure 1). 

SNUDLE was designed with a purposeful focus on lowering construct-irrelevant barriers 
to science learning.*° Text-to-speech technology is built in to the notebook interface with real- 
time highlighting to support simultaneous access to auditory and visual processing, as well as 
word-by-word English-to-Spanish translation, keyboard-accessible actions, and a multimedia 
glossary to provide just-in-time support for vocabulary use and development. These features help 
the many students whose literacy skills are not commensurate with the reading and writing 
proficiency-dependent materials,*’*° for those whom proficiency in English is a barrier, and 


others who would learn more effectively through use of built-in accessibility features. 


SNUDLE also leverages contextual support that is intended to develop and reinforce 
effective science learning behaviors. Pedagogy is built into the interface design itself, guiding 
students and teachers in the process of active science learning and the effective use of science 
notebooks. For instance, students are prompted to think about making direct reference to their 
data and observations and to use relevant vocabulary from their inquiry experiences. 

In addition to the student-facing interface, features of the teacher’s role in SNUDLE 
facilitate active science learning. Teachers are prompted and supported to provide feedback that 
may include corrective information, alternative strategies, information to clarify ideas, or 


encouragement to engage in the scientific process.” 


Methods 

This was a fully powered randomized controlled trial efficacy study of fourth grade 
students in elementary schools in a large urban school district. The study sample was comprised 
of 683 students (372 intervention, 311 comparison) across seven elementary schools. All fourth 
grade students participating in inclusive general education science classes were eligible to 
participate in the study. From the student’s school record, we obtained data relevant for the 
determination of eligibility for special services for disabilities as well as English learner status 
and language spoken at home. To further describe the sample, we collected demographic and 
academic characteristics. 

This study was based on classroom-level randomization. Twenty-nine fourth grade 
teachers were recruited to participate for 2 years of data collection and were randomized to the 


SNUDLE intervention group or business as usual comparison group. 


Measures 

Curriculum-based unit tests. Researcher-developed unit tests were derived from 
assessment items from STEMscopes, the school district’s curriculum. These unit tests were used 
as a measure of academic achievement measures closely aligned with the curriculum content. 
The curriculum developers had categorized the items by the four levels of Bloom’s Taxonomy": 
Understand, Apply, Analyze, and Evaluate. Because SNUDLE seeks to provide opportunities to 
improve higher level science thinking, the items we selected predominantly focused on 
“Analyze” and “Evaluate” questions. One of the nine quizzes was dropped from analysis because 
a natural disaster caused school closure at the beginning of the study that interrupted teaching 
and quiz administration. The number of correct responses across the remaining eight end-of-unit 
quizzes served as a proximal outcome measure. The standardized Cronbach coefficient alpha for 
the STEMscopes unit tests was .88. 

Measures of Academic Progress (MAP®). For a broader measure of science knowledge, 
we administered the Northwest Evaluation Association’s MAP test of science at the end of Year 
1 data collection. The MAP science test is a formative measure that covers domains of earth, life, 
and physical sciences. It is a computerized adaptive assessment consisting of 50 multiple-choice 
items with four or five options. In the Northwest Evaluation Association’s item development, all 
items match the assessable sections of a set of academic content standards both in breadth of 
content and depth of knowledge. MAP tests have been validated to link to content standards in 
all 50 states and have excellent technical characteristics.’ 

Motivation for Science Inventory (MFS). The MFS is an 18-item survey consisting of 


subscales for the following constructs: self-efficacy, interest, desire for challenge, and comfort 


using computers. 7” Reliability of the MFS survey is 0.85; for the experimental sample, it was 
0.89.73 
Results 

Student Characteristics 

Of the 683 students consented to participate in the study, 649 of them had complete 
demographic information. Table 1 shows the demographic characteristics of the intervention 
group and comparison group. The two groups generally were balanced across conditions on most 
demographic characteristics, except that the treatment group had higher a proportion of African 
American students and a lower proportion of Hispanic students than the control group. The 
treatment group also had a higher proportion of students whose home language was English than 
the control group. 

Student Assessment Scores 

Table 2 presents the student assessment scores at baseline and posttest for each of the 
outcome measures: the MAP, four subscales of the MFS, and the sum of the eight quiz scores 
derived from the STEMscopes curriculum. Students with both pretest and posttest were included 
in the table. 

Attrition 

For the MAP outcome, the treatment group attrition rate was 9%, the control group 
attrition rate was 15%, and the differential attrition rate was 6%. For the MFS outcome, the 
treatment group attrition rate was 21%, the control group attrition rate was 28%, and the 
differential attrition rate was 7%. For the total quiz score outcome, the treatment group attrition 


rate was 8%, the control group attrition rate was 13%, and the differential attrition rate was 5%. 
group 


Teachers and students who joined the schools after randomization were not included in the 
sample or analysis. 

Main Impact Analysis 

Primary estimates of the intervention effect were derived from the intent-to-treat 
analyses. Regardless of the level of implementation, these analyses compared all students in 
treatment teachers’ classrooms with their peers in control teachers’ classrooms. A two-level 
hierarchical linear model (HLM) was performed to account for students nested in teachers’ 
classrooms. Dependent variables were the MFS, MAP, or quiz score. Independent variables were 
a constant, a pretest score on the same outcome measure or STAAR reading score when pretest 
on the same outcome measure was not available, demographic characteristics, and treatment 
indicator. Table 3 presents impact estimates from the listwise deletion HLM. Two-level HLM 
models were conducted in which Level | was the student level and Level 2 was the school level. 
Table 3 also shows the Hedge’s g effect size on each outcome among the whole sample. The 
treatment group did not differ from the control group on any of the outcomes after controlling for 
pretest and baseline demographic characteristics. 

Moderator Analysis 

We examined the effect of the intervention among students with disabilities, identified by 
their having an Individualized Education Plan (Table 3). These HLM impact analyses only 
included students with disabilities. We found a consistent positive treatment effect on all four 
MEFS subscales (Efficacy: g = 1.67, p < 0.01; Interest : g = 1.54, p < 0.05; Desire for challenge: g 
= 1.93, p < 0.05; Comfort using computer: g = 1.46, p < 0.01), and total unit quiz score (g = 1.32, 


p<0.01). 


Significance 

The preliminary findings from the first year of a 2-year efficacy study provide evidence 
of SNUDLE’s efficacy, particularly for struggling students with learning disabilities. In 
evaluating SNUDLE’s efficacy, this study works to address the pressing need to expand the 
evidence base for universally designed approaches to support struggling students in active 
engagement with science at the elementary school level. 

In practice, implementation of UDL and UDL-designed tools is not uniform, even by a 
single teacher. Rather than a strict set of rules, UDL is a flexible framework that researchers and 
professionals can use to guide teaching methods, classroom structure, and the design of 
curriculum, assessment, and classroom materials. In addition, a foundational tenant of UDL is 
recognizing and respecting variability in learning and learners.*> This variation in 
implementation and use creates challenges in conducting rigorous examinations of UDL-based 
interventions and some skepticism about UDL’s demonstrable efficacy.*° For example, a 
systematic review of the efficacy of UDL-based interventions in 13 studies*’ found significant 
variation in both the effect sizes of findings in the studies and the way UDL principles were 
translated into the interventions examined. Another systematic review found that while UDL can 
be a useful guide for increasing accessibility and designing flexible teaching practices and 
learning environments, more rigorous academic research is needed to demonstrate its impacts.*® 

These systematic reviews identify several areas of focus for future research that are 
contributed in our efficacy trial: rigorous study designs and procedures, opportunities to 
operationalize UDL features, explicit connections between UDL guidelines and the SNUDLE 
product and practice features, and an examination of the effects of UDL on students with 


disabilities.*”** In this efficacy study, we are examining the measurable outcomes of a specific 
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UDL-based tool in a classroom context rather than seeking to provide evidence for the efficacy 
of UDL as a framework. Even so, the incompleteness of the UDL evidence base due to inherent 
implementation variation and flexibility makes this study a critical contribution to the research 
base for a promising framework that seeks to make learning more accessible and engaging for all 


students. 
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Table 1. Baseline demographic characteristics of students 


Student characteristics Treatment Control 
(n=362) (n=287) 
Male 48% 47% 
Race/Ethnicity 
White 5% 4% 
African American 42% 26% 
Asian 13% 13% 
American Indian/Alaskan 1% 1% 
Hispanic 39% 55% 
Free or reduced lunch status 82% 83% 
Dual Language Learner status 25% 26% 
Home language 
English 50% 35% 
Spanish 32% 47% 
Other 18% 17% 
Individualized Education Plan T% 5% 
504 plan 3% 3% 


Note. Treatment sample size is 372 and control sample size is 311. 362 out of the 372 treatment students have no 
missing data on any demographic variables. 287 out of the 311 control students have no missing data on any 


demographic variables. 


Table 2. Pretest and posttest scores by treatment assignment 


Student assessments Treatment Control 

N Mean SD N Mean SD 
STAAR reading used as pretest for 338 | 1412.04 | 158.56 | 264 | 1427.15 | 154.09 
MAP Posttest 
MAP RIT Posttest 338 | 198.09 | 11.86 | 264 | 198.70 | 10.67 
MFS — Efficacy Pretest 296 | 3.35 0.63 225 | 3.34 0.54 
MFS — Efficacy Posttest 296 | 3.45 0.55 225 | 3.39 0.52 
MFS — Interest Pretest 295 | 3.15 0.41 224 | 3.13 0.36 
MFS — Interest Posttest 295 | 3.12 0.45 224 | 3.14 0.43 
MFS- Desire for challenge Pretest 294 | 3.06 0.47 225 | 3.09 0.40 
MFS- Desire for challenge Posttest 294 | 3.00 0.47 225 | 3.09 0.41 
MFS - Comfort using computer Pretest | 295 | 3.34 0.59 225 | 3.30 0.54 
MFS - Comfort using computer 295 | 3.16 0.60 225 | 3.14 0.58 
Posttest 
STAAR reading used as pretest for 343 | 1413.20 | 157.87 | 271 | 1427.89 | 154.03 
total quiz score 
Total quiz scores 343 | 40.38 15.52 | 271 | 40.49 17.05 


Note. STAAR = State of Texas Assessments of Academic Readiness; MFS = Motivation for Science 
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Table 3. Estimated treatment impact among whole sample and students with disabilities 


The Whole Sample Students with Disabilities 
Outcomes B SE Effect Size | B SE Effect Size 
MAP 0.62 1.29 0.05 -1.41 3.27 -0.11 
MFS - Efficacy 0.07 0.04 0.13 1.25** 0.40 1.67 
MFS -Interest 0.04 0.03 0.09 0.72* 0.31 1.54 
MFS -Desire for challenge 0.02 0.04 0.04 1.28* 0.47 1.93 
MFS -Comfort using computer | 0.05 0.05 0.08 0.84** 0.27 1.46 
Total quiz 3.13 3.34 0.19 16.44** | 4.64 1.32 


*p < .05, **p < .01 
Note. Benjamin-Hochberg multiple comparison adjustment was not run because treatment impact was not significant. 
The HLM controls for pretest, gender, race, free or reduced lunch status, dual language learner status, home 


language, and IEP status. 
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